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i  .  1  3ACKGR07ND 

Test  design  and  planning  for  modern  Command,  Control,  Communications  and 
Intelligence  IC’D  systems  is  becoming  an  increasingly  complex  task.  More 
sophisticated  systems  are  requiring  mere  complex  testing,  in  an  environment 
with  more  demands  and  tighter  budget  constraints.  At  the  heart  o:  this 
problem  is  the  increasing  use  of  electronics,  computers,  and  communications  in 
systems  with  large,  distributed  architectures.  Because  of  the  exploitation  of 
advanced  technology  even  for  relatively  small  test  items,  more  testing  is 
required  today  than  previously.  The  increased  test  load  involves  more  types 
of  testing,  such  as  software  tests  for  the  embedded  microcomputer,  and  more 
automated  test  instrumentation . 

The  indirect  effects  are  nearly  as  great,  if  not  larger.  Modern 
technology  imposes  new  demands  on  the  tester  indirectly  through  more  complex 
security,  safety,  budget,  and  environmental  considerations.  The  consequence 
of  this  situation  is  that  testing  is  rapidly  reaching  a  point  where  the 
expertise  required  is  too  great  for  any  one  individual  to  handle  effectively. 
By  the  time  expertise  is  acquired  in  any  one  area,  requirements  are  likely  to 
change,  or  the  individual  may  retire,  leave,  or  transfer  out  of  the 
organization . 

The  7.3.  Army  Electronic  Proving  Ground  t'JSAEPG)  has  positioned  itself  to 
alleviate  some  of  the  problems  faced  by  today’s  test  officer,  by  exploiting 
some  of  the  very  technology  which  is  partly  responsible  for  this  dilemma: 
artificial  intelligence  (AI)  and  the  much  improved  microcomputer.  Previous 
investigations  at  USAEPG,  sponsored  by  the  Department  of  Defense  (DoD) 

Software  Technology  for  Adaptable,  Reliable  Systems  (STARS)  program  (reference 
1),  identified  some  aspects  of  AI  which  were  sufficiently  mature  to  insert  m 
test  toois.  One  of  these  technologies,  AI  expert  systems,  was  explored  in 
depth . 

Prototype  expert  systems  were  developed  to  demonstrate  capabilities  and 
potential  benefits.  One  of  the  first  systems  built  was  (and  currently  is) 
used  to  assess  the  suitability  of  proposed  applications,  as  some  problems  are 
best  addressed  with  conventional  software  techniques.  After  the  expertise  was 
established  to  build  small  expert  systems,  a  workshop  was  conducted.  This 
workshop  served  to  link  up  the  knowledge  engineers  (AI  experts  skilled  at 
using  AI  software  tools)  with  domain  experts  (the  people  who  understand  the 
problems  and  are  skilled  at  providing  solutions) .  The  workshop  approach 
provided  the  attendees  and  their  managers  with  the  familiarity  which  is  vital 
to  the  successful  use  of  any  new  technology. 

Many  ideas  for  expert  systems  were  produced  as  a  res  ’t  of  the  workshop, 
and  a  few  of  the  workshop  projects  evolved  into  larger  prototype  versions. 
Perhaps  of  more  importance  than  the  actual  systems  developed,  were  some  of  the 
lessons  learned.  While  previous  evpert  systems  had  bee--  hosted  on  large 
minicomputers  or  specialized  AI  machines,  the  toois  u*ed  m  the  workshop  were 
meant  for  both  development  and  use  on  microcomputers.  Once  the  feasibility  of 
building  expert  systems  for  the  widely  available  microcomputers  was 
established,  the  potential  usefulness  and  possible  applications  of  this  AI 
technology  increased  dramatically.  Given  this  rather  fortuitous  situation, 
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USAE? 3  was  afforded  the  opportunity  to  exploit  AI  technology  for  solving  the 
everyday  problems  of  the  test  officer. 

1.2  OBJECTIVE 

1.2.1  Applications . 


The  objective  of  this  investigation  was  to  provide  the  test  officer  with 
automated  support  tools  by  inserting  AI  technology  m  appropriate 
applications.  Objectives  for  the  development  of  these  tools  included: 


a.  Orientation  toward  the  test  officer  as  primary  user. 

b.  Wide  usability  to  satisfy  the  needs  of  the  approximately  100  test 
officers  at  the  USAEPG. 


c.  Heady  availability  (microcomputer  based; . 

d.  Seduction  in  time  to  perform  a  given  task  and/or  improved  quality  of 
the  result. 

e.  Education  of  the  user  (test  officer)  in  addition  to  merely  providing 
a  solution. 

1.2.2  Neural  Computing. 

An  ancillary  goal  was  to  maintain  awareness  of  developments  m  the  AI 
arena  which  may  prove  useful  in  f uture  ■  phases  of  the  investigation.  One  area 
of  Ai ,  which  was  examined  in  some  detail,  was  the  field  of  neural  computing. 

1.2.3  Testing. 


-  Finally,  as  testers,  a  goal  of  every  investigation  is  to  identify  any 
test  methodology  requirements  which  surface.  Theyed-ore,  another  objective  was 
to  assess  the  adequacy  of  current  test  methodology  for  the  test  and  assessment 
of  systems  containing  AI . 


1.3  SUMMARY  OF  PBOCEDUBES 


1.3.1  Applications . 


wessons  learned  from  earlier  work  on  expert  system  development  were 
applied  to  restructure  the  proposed  approach.  Rather  than  develop  a  single 
test  officer  tooi  on  the  one  available  AI  machine,  an  approach  more  in 
consonance  with  the  objectives  was  established.’  This  approach  called  for  the 
development  of  a  numb""  of  small  tools,  with  a  greater  overall  probability  of 
success  than  that  of  a  single  large  tool.  The  development  of  smaller  tools 
hosted  on  microcomputers  also- provided  a  more  flexible  means  of  adjusting  to 
various  constraints  /  while  also  benefiting  from  the  experience  gained  during 
previous  efforts.  ’ 


The  resulting  approach  consolidated  efforts 
workshop,  by  furthering  development  of  the  Test 
Environmental  Impact  Assessment  (EVA)  systems. 


of  the  expert  systems 
Plan  Drafter  (T?D)  and 
From  this  initial  base 


new 


’  - -> 


ideas  were  developed  :r.  *  he  areas  cf  met  eoro  leg  i  cal  supper*,  budget,  ar.d 
security.  Systems  addressing  these  problem  domains  were  developed  using  the 
workshop  methodology:  problem  domain  experts  and  knowledge  engineers  were 
paired  to  develop  Al-based  test  officer  support  tools. 

1.3.2  N'eural  Computing. 

An  examination  of  emerging  AI  technology  led  to  the  further  investigation 
of  neural  computing.  While  the  foundations  for  neural  computing  were 
established  when  AI  was  still  in  its  infancy,  only  recently  has  the  technology 
matured  to  the  point  where  application  has  been  possible  outside  the  research 
milieu. 

1.3.3  Testing. 


Finally ,  the  issue  of  testing  AI  systems  was  explored  to  a  limited 
extent;  first,  because  the  development  of  expert  system-based  tools  required 
an  ir.-house  test  philosophy;  and  second,  because  test  items  employing  AI 
technology  are  likely  to  appear  m  the  near  future.  One  initiative  m  this 
area  was  to  develop  interest  within  the  AI  community  m  addressing  testing 
1 ssues . 

1.4  SUMMARY  OF  RESULTS 
1.4.1  Applications . 

A  number  of  AI  expert  systems  were  developed  to  aid 
performing  duties  associated  with  testing.  With  respect 
objectives  in  section  1.2.1,  these  systems  possessed  the 
characteristics  to  the  extent  noted. 

a.  The  knowledge  domains  of  the  expert  systems  centered  on  areas  of 
expertise  of  which  an  experienced  test  officer  would  be  cognisant,  but  not 
necessarily  an  expert.  In  other  words,  a  test  officer  might  be  familiar  with 
certain  test  planning  requirements,  such  as  drafting  test  plans  or  examining 
potential  environmental  impacts,  but  would  still  require  considerable 
consultation  with  a  domain  expert  to  satisfy  the  requirements  for  a  new  test. 
The  prototype  systems  built  during  this  phase  of  the  investigation  were 
intended  to  assist  test  officers  by  providing  the  preliminary  advice  normally 
obtained  from  the  domain  expert  curing  test  planning. 

b.  Most  of  the  systems  developed  are  still  m  the  evaluation  phase  ar.d 
therefore  have  been  installed  on  a  limited  number  of  computer  systems.  The 
EVA  ar.d  Meteorological  (MET)  expert  systems  are  presently  installed  on  i?ven. 
microcomputers  located  throughout  the  various  organisations;  while  other 
expert  systems,  such  as  the  TPu,  are  m  ‘■ailed  ir.  the  expert’s  office.  Ail  of 
the  domain  experts  have  the  expert  systems  installed  on  their  personal 
machines,  for  use  and  evaluation  whenever  a  test  officer  consulting  them  has 
not  previously  used  one  of  the  systems  for  a  particular  test  project.  This 
arrangement  also  allows  the  experts  to  verify  system  performance  and  recommend 
changes,  to  address  unique  requirements  or  to  adjust  for  inadequately  covered 
areas  of  the  problem  domain. 


the  test  officer  m 
to  the  application 
f  ol lowing 
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Ail  of  the  s  v  3 1  e  ms  were  targeted  for  the  micro 
'JSAEPG.  Because  of  one  different  conf igurat ions  ir.  use.  sore 
exist  as  to  which  functions  can  be  used  while  still  retaining 
with  a  majority  of  the  microcomputer  base.  Primarily  these  co 
concerned  disk  and  memory  sine,  graphics  capabilities,  and  har 
accelerators  for  floating  point  operations.  Prom  a  practical 
little  functionality  has  been  lost  m  conforming  to  the  minima 
Only  ore  expert  system,  the  TPD ,  requires  licenses  for  support 
functions;  and  this  is  because  of  non-AI  functional  requiremer, 
extensive  data  base  management  capability  and 
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d.  An  assessment  of  time  savings  cr  improved  reality,  due  to  the  use  of 
expert  system  aids,  can  only  be  done  qualitatively,  since  all  0:  the  systems 
are  ;ust  now  being  evaluated  using  actual  test  project  parameters.  Projected 
savings  are  considerable  m  seme  cases;  the  TPD  should  reduce  the  time  to 
prepare  test  plans  from  several  days  to  a  few  hours.  Other  expert  systems, 
such  as  the  EVA,  have  demonstrated  the  benefit  0:  providing  preliminary 
assistance  m  what  can  be  a  complex  task.  At  a  minimum,  they  can  enforce  the 
collection  and  consistent  presentation  0:  necessary  documentation.  Ail  of  the 
systems  have  demonstrated  the  ability  to  retain,  and  even  combine,  expertise 
from  human  domain  experts. 

e.  The  present  suite  of  support  tools  ail  serve  to  tram  the  test 
officer  to  seme  degree.  After  running  the  expert  systems  a  few  times,  the 
user  begins  to  understand  which  parameters  are  s.gmficant  for  given 
situations.  Also,  ail  of  the  systems  provide  an  on-lme  ‘help"  function  to 
inform  the  user  of  the  nature  of.  and  appropriate  response  to.  the  various 
queries  encountered.  Although  the  expert  system  shells  have  an  expiar.ati  :r. 
capability,  in  terms  of  the  knowledge  base  rules  used  to  reach  a  conclusion, 
this  feature  is  not  called  upon  by  the  average  user.  Most  of  the  advice 
offered  by  the  systems  provides  both  the  necessary  action  ar.d  the  reason  for 
the  action;  e.g.,  use  of  incendiary  devices  requires  filing  a  fire  plan  with 
the  post  fire  marshal. 


1.4.2  Neural  Computing. 

Neural  computing  technology  was  examined  sufficiently  to  gam  familiarity 
with  the  fundamentals,  identify  potential  testing  applications,  and  suggest  a 
course  of  action  for  further  investigation. 

1.4.3  Testing . 

Test  technology  for  AI  expert  systems  is  almost  nonexistent.  Rudimentary 
orocedures  were  established  for  in-house  use,  with  the  hope  that  these  r.av 
lead  to  adding  a  more  formal  test  dimension  to  expert  system  development 
acvities.  Not  much  progress  has  been  made  by  others  m  the  field  either, 
although  quite  a  lot  of  interest  m  testing  issues  was  expressed  by  atter.cees 
at  an  expert  system  workshop  heid  in  conjunction  with  the  1963  annual  American 
Association  for  Artificial  Intelligence  t  AAAI i  conference.  Ail  indications 
suggest  that  future  conferences  will  continue  to  support  this  attempt  *0 
provide  test  technology  corresponding  to  advances  m  AI  technology. 
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ANALYSIS 


I  .  5  .  I  Appi  leathers  . 

The  development  of  various  expert  systems  to  aid  the  test  officer 
demonstrates  the  applicability  of  this  AI  technology  to  problems  ir.  the 
testing  environment.  The  systems  are  still  being  evaluated,  and  will  probably 
continue  to  evolve  to  support  more  of  the  domain  knowledge.  However,  the 
potential  utility  of  the  development  methodology  used  is  readily  apparent. 
Besides  the  obvious  benefits,  such  as  retained  knowledge  and  combined 
expertise  of  multiple  experts,  this  methodology  showed  the  feasibility  of 
developing  anc  using  expert  system  technology  with  existing  microcomputer 
resources.  In  addition,  improved  productivity  and  quality  of  work  can  be 
expected  from  both  the  test  officers  and  the  domain  experts.  With  increasing 
emphasis  on  efficiency  being  dictated  by  leaner  budgets,  these  last  two 
considerations  may  overshadow  other  potential  advantages  of  AI . 

The  prototype  systems  developed  for  the  investigation  addressed 
individual  problem  domains  within  the  testing  milieu.  Since  many  of  these 
domains  share  commonality  of  information  (here  meaning  both  data  and 
knowledge)  about  test  resources,  techniques,  and  requirements ,  a  broader 
analysis  of  test  support  requirements  is  appropriate.  An  early  examination  of 
the  testing  infrastructure,  with  subsequent  incorporation  of  global 
requirements  into  a  supporting  structure  (i.e.,  data  bases,  networks, 
geographic  information  systems,  and  standard  information  elements),  could 
eventually  lead  to  an  integrated  set  of  cooperating  support  tools. 


1 . 5 . 2  Neural  Computing. 

Neural  computing  technology  has  evolved  to  a  point  of  limited  practical 
applicability,  much  as  expert  system  technology  did  a  few  years  ago.  As  with 
the  early  commercialization  of  expert  systems,  neural  networks  will  probably 
suffer  from  being  oversold  and  applied  indiscriminately  with  overoptimisti c 
expectations.  However,  like  expert  3ystem  technology,  neural  computing 
methods  will  most  likely  prove  to  be  practical  for  small,  well-defined  problem 
areas.  For  many  applications,  this  will  mean  that  the  neural  network  will  be 
merely  a  component  of  a  larger  system  comprised  of  conventional  software 
(procedural  code,  data  bases,  and,  by  then,  expert  systems).  The  treatment  of 
large  problems,  with  specialized  hardware  to  allow  reasonable  execution  times, 
will  best  be  left  to  the  research  environment  tor  the  near  term.  There  is 
little  doubt,  however,  that  this  technology  will  find  its  niche  in  system 
developers'  toolkits. 

1.5.3  Testing . 


The  application  of  AI  has  not  awaited  the  development  of  adequate  test 
methodologies.  This  has  occurred  with  expert  systems,  and  will  also  be  the 
case  for  neural  networks.  Premature  use  of  such  technology  presumes  the 
existence  of  benefits  which  outweigh  the  risks  incurred  by  l.ack  of  formal 
testing  techniques.  One  obvious  issue  which  arises  is  a  comparison  o f  the 


performance  of  A,  components  to  similar  functionality  (if  possioie)  provided 
by  conventional  technology.  In  partial  defense  of  this  use  of  technology  just 
out  of  the  laboratory,  it  should  be  realized  that  more  conventional  software 
techniques  are  routinely  used  without  mature  test  methodologies  (e.g.. 
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distributed  data  bases),  or  at  least  without  methods  which  are  cotr. 
comprehensive  and  cost  effective. 

1.6  CONCLUSIONS 

1.5.1  Apsl ications . 


The  investigation  demonstrated  the  viability  of  Al-based  automated  tools 
to  assist  the  test  officer.  This  was  accomplished  with  existing  microcomputer 
resources,  which  increased  the  availability  whiie  minimising  implementation 
costs.  Further  validation  of  this  microcomputer-based  expert  system 
development  methodology  would  require  completion  of  the  evaluation  phase,  with 
subsequent  distribution  of  the  systems  throughout  the  organization.  Solviz 
the  infrastructure  problem  in  conjunction  with  development  of  expert  system 
is  too  large  an  effort  to  be  absorbed  by  follow-on  phases  of  this 
investigation.  However,  some  consideration  should  be  given  to  the  eventual 
integration  of  the  support  tools. 

1.6.2  Neural  Computing.  ~ 

Neural  computing  appears  to  offer  substantial  benefits,  especially  if 
viewed  as  merely  another  AI  technique  to  be  used  in  conjunction  with  other 
methods.  Both  the  advantages  of  neural  networks  in  test  tools  and  their 
potential  use  in  test  items  make  it  prudent  to  maintain  awareness  of 
developments  in  this  area. 

1.6.3  Testing . 

Since  systems  are  already  being  developed  which  employ  expert  system 
technology,  it  is  imperative  that  adequate  te3t  techniques  be  developed 
immediately.  Test  procedures  and  test  issues  should  be  established  for  the 
various  types  of  AI  technology,  different  methods  of  implementation,  and 
typical  operational  characteristics. 

1.7  RECOMMENDATIONS 

1.7.1  Aopl ications . 

Further  investigation  is  recommended  in  the  following  areas: 

a.  The  tool  base  developed  during  the  initial  phase  should  continue 
through  the  evaluation  phase  to  further  validate  the  results.  Widespread 
distribution  ar.d  operational  considerations  should  be  addressed,  ar.d 
maintenance  of  the  knowledge  bases  should  be  investigated.  Further 
development  of  test  officer  support  tools  should  also  factor  in  infrastructure 
requirements  to  the  extent  possible. 

b.  A  s»  ."'arate  investigation  i3  required  to  analyze  the  requirements  for 
establishing  and  maintaining  an  automated  testing  infrastructure. 


M  OvX 


1.7.2 


Neural  Computing. 

Neural  computing  3hould  be  investigated  further,  either  as  a  separate 
investigation,  cr  by  the  application  of  increased  resources  to  follow-on 
efforts  of  this  investigation . 

1.7.3  Testing . 


An  investigation  is  required  to  develop  test  procedures  for  AI . 
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SECTION 
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INVESTIGATION 


2.1  APPLICATION  OF  Al 

One  of  the  spin-offs  of  AI  research  has  been  the  development  of  tools 
known  as  expert  system  shells  to  assist  m  the  construction  of  rule-based 
expert  systems.  These  expert  system  shells  aliow  a  knowledge  engineer  to 
codify  logical  inferences  (rules)  about  a  given  domain,  and  process  the 
resulting  knowledge  base  to  provide  expertise  to  the  user.  This  investigation 
examined  the  potential  of  applying  this  technology  to  assist  the  test  officer 
by  developing  various  Al-based  support  tools. 

2.1.1  Accroach . 


Historically,  most  expert  systems  have  been  developed  by  a  team 
consisting  of  AI  experts  and  domain  experts,  although  in  a  few  instances  the 
domain  experts  acted  as  their  own  knowledge  engineers.  It  is  the  job  of  the 
knowledge  engineer  to  obtain  knowledge  about  a  particular  domain  through 
consultation  with  one  or  more  experts,  documented  information,  or  some 
combination  of  these  sources.  This  knowledge  is  then  incorporated  into  an 
automated  tool  which  uses  this  expertise  in  solving  problems  within  the 
domain.  Expert  system  shells  have  considerably  eased  the  task  of  developing 
expert  system  tools,  by  providing  a  means  to  enter  and  exercise  logical  rules 
about  a  given  domain.  In  this  sense,  these  tools  are  similar  to  the 
capability  offered  by  conventional  software  language  compilers/ interpreters . 

Recent  developments  in  expert  system  shells  have  resulted  m  a  number  of 
tools  which  are  relatively  easy  to  use,  and  do  not  require  extensive 
programming  skills  such  as  those  normally  associated  with  using  symbolic 
programming  languages.  These  shells  have  made  it  possible  for  some  domain 
experts  to  build  expert  systems  without  assistance.  However,  knowledge 
engineering  encompasses  more  than  merely  entering  rules  in  the  proper  format. 
One  need  for  knowledge  engineers  arises  when  the  complete  system  includes 
conventional  software  components.  In  these  situations  the  knowledge  engineer, 
who  is  usually  skilled  in  software  development,  can  develop  a  design  which 
will  satisfy  all  of  the  system  requirements. 

USAEPG  used  the  knowledge  engineer/domain  expert  team  concept  in 
developing  support  tools  for  the  test  officer.  Part  of  the  approach  involved 
conducting  an  expert  system  workshop  to  familiarize  personnel  with  the 
technology  and  to  solicit  ideas  for  further  development.  The  workshop 
included  six  teams,  each  of  which  built  a  small  expert  system  as  a  class 
exercise.  Of  these,  three  systems  were  selected  for  development  of  a 
prototype  system,  based  on  a  raanagert'"t  review  of  the  class  projects.  Other 
support  tools  were  conceived  after  the  workshop  concepts  had  evolved  and  idea 
:or  potential  applications  became  more  apparent.  One  side  benefit  was  the 
exposure  of  both  management  and  test  experts  to  the  capabilities  and 
limitations  of  expert  systems. 


Applications  proposed  for  test  officer  support  *ools  were  further 
screened  by  an  existing  tool  which  determines  the  probable  success  of  a 
proposed  system  by  analyzing  various  parameters  of  the  project.  This  system, 
the  Expert  System  Selector  (E S - ) .  is  itself  an  expert  system.  ESJ  examines 
such  factors  as  the  availability  of  expertise,  and  supporting  development  ar.d 
runtime  tools;  and  the  suitability  and  feasibility  of  an  expert  system 
solution.  It  then  provides  a  qualitative  score  of  the  overall  success 
potential.  Proposed  concepts  were  required  to  be  sufficiently  well  defined  to 
be  graded  by  the  ES-  before  being  considered  for  development.  This  approach, 
in  fact,  was  used  to  screen  ideas  for  the  initial  workshop,  and  was 
responsible  for  the  elimination  of  what  would  have  been  poorly  suited  or 
overly  ambitious  suggestions. 

2.1.2  Environment . 


The  computing  resources  of  USAEPG  are  considerable,  and  include  a  variety 
of  mainframe,  mini,  micro,  and  special-purpose  computers.  The  sophisticated 
machines  tailored  to  AI  applications  are  not  readily  available  to  the  test 
officer,  however.  For  administrative  functions,  including  test  planning 
activities,  microcomputers  are  the  primary  computing  resource.  Earlier  AI 
efforts  at  UEAEPG  relied  on  large  minicomputers  or  symbolic  processors. 
However,  the  emergence  of  microcomputer-based  expert  system  development  tools 
was  noted,  and  some  shells  were  acquired  to  determine  the  practicality  of  AI 
systems  targeted  for  the  smaller  machines.  Earlier  products  had  been  too 
slow,  many  impractical,  even  when  implemented  on  the  larger  machines. 

2.1.3  Characteristics  of  Testing  Organization. 

USAEPG,  like  the  other  subordinate  elements  of  the  U.S.  Army  Test  and 
Evaluation  Command  (TECOM) ,  assigns  action  officers  to  oversee  the  activities 
associated  with  test  directives.  These  test  officers  must  wear  a  number  of 
hats  in  performing  their  duties.  Besides  test  planning,  the  test  officer  is 
responsible  for  monitoring  actual  test  conduct,  and  of  course  analyzing  and 
reporting  the  results.  With  test  items  increasing  in  complexity  due  to  the 
increased  use  of  electronics,  computers,  and  communications,  more  types  of 
tests,  of  a  greater  variety,  must  be  performed  in  today’s  testing  environment. 
This  would  be  sufficiently  challenging  without  the  additional  burden  of 
reduced  budgets  and  increased  documentation  requirements.  At  USAEPG  alone, 
approximately  100  personnel  are  designated  as  test  officers,  with 
responsibility  for  conforming  to  all  of  the  appropriate  directives, 
regulations,  and  guidelines  without  losing  sight  of  the  primary  mission. 

2.1.4  Knowledge  Infrastructure. 

Expert  systems  designed  to  provide  advice  and  assistance  m  policy  ar.d 
procedure  within  the  context  of  a  large  organization  require  at  least  two 
types  of  knowledge.  The  first  type,  knowledge  of  the  domain  in  which  the 
system  is  to  advise  and  assist,  is  termed  domain  expertise,  and  is  the  object 
of  the  knowledge  acquisition  effort  as  commonly  described  m  AI  literature. 
The  second  type  involves  information  concerning  the  administrative, 
organizational,  and  regulatory  environment  within  which  the  expero  and  system 
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mus *  operate.  This  knowledge  bears  a  relationship  to  domain  expertise 
analogous  tc  that  between  a  given  industrial,  business,  or  administrative 
facility  and  the  complex  suite  of  transportation,  communication,  and  utility 
facilities,  standards,  and  administrative  provisions  necessary  to  their 
operation.  The  term  infrastructure  has  been  borrowed  from  the  economic  and 
geopolitical  literature  to  describe  this  knowledge. 

Within  USAEPG,  requisite  information  is  widely  available,  but  from  a  wide 
variety  of  sources.  At  this  time,  there  is  no  central  point  for  maintenance 
of  or  access  to  this  information.  Examples  of  the  types  of  information 
involved  include: 

a.  Organisation ,  Mission,  and  Functions  manual. 

b.  Organization  charts  for  USAEPG  and  related  organizations. 

c.  Lists  of  DoD,  Army,  AMC ,  TECOM,  and  USAEPG  publications. 

d.  Lists  of  non-DoD  and  industry  standards  and  related  publications. 

e.  Project/topic/system  point-of-contact  lists. 

f.  Standard  distribution  lists. 

g.  Acronym/abbreviation  lists,  etc. 

2.2  AI  EXPEH?  SYSTEM  APPLICATIONS 

An  AI  expert  system  development  methodology  for  test  officer  support 
tools  was  synthesized  from  the  lessons  learned  from  previous  projects,  AI 
technology  capabilities,  computer  resource  availability,  elements  of  the 
testing  environment  and  test  officer  duties,  and  characteristics  of  the 
domains  deemed  suitable.  This  resulted  in  an  approach  similar  to  those  used 
by  industry  for  smaller  AI  applications: 

a.  Acquisition  of  microcomputer  development  tools  and  related  personnel 
skills. 

b.  Identification  of  suitable  applications. 

c.  Teaming  of  a  knowledge  engineer  and  domain  expert(s). 

d.  Prototyping  and  iterative  development  of  the  expert  systems. 

The  result  o'  implementing  this  methodology  was  the  generation  of  a 
number  of  small  expert  systems  which  address  problems  encountered  by  the  test 
officer.  Most  of  the  systems  deal  with  requirements  during  the  planning 
phases  of  a  test.  This  is  not  an  indication  that  expert  systems  are  not 
suitable  for  test  conduct  or  reporting  activities,  but  probably  does  reflect 
the  greater  stability  and  better  defined  nature  of  the  planning  stage.  That 
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is,  lest  plans  and  environmental  documenta 
of  other  variations  in  the  test  activity. 


on  are  always  required,  re* 


The  prototype  test  officer  support  toois  built  during  the  investigation 
are  described  below.  For  each  system,  the  purpose  and  goals,  domain, 
requirements,  description,  design  characteristics,  benefits,  and  status  are 
briefly  described. 


2.2.1  Test  Plan  Drafter. 


2.2.  1.1  Purpose/Goals .  The  near-term  goal  of  the  TPD  is  to  automate  the 
current  manual  assembly  of  boilerplate  for  an  initial  draft  of  a  detailed  test 
plan  'DTP).  This  is  a  time-cor.sumir.g  effort  consisting  of  much  cut-and-paste 
work  from  old  test  plans,  but  little  real  intellectual  effort.  It  is  intended 
to  result  in  a  strawman  version  for  distribution  to  specific  subtest  domain 
experts  for  further  editing. 

A  longer  term  goal  of  the  effort  is  to  create  a  prototype  knowledge 
infrastructure,  i.e.,  a  structure  for  centralized  maintenance  of  both  specific 
information  pertaining  to  a  given  test  and  general  information  needed  m  test 
planning . 

2.2. 1.2  Domain/Expertise .  The  initial  knowledge  acquisition  for  T?D  involved 
determining  the  structure  and  composition  of  a  DTP.  This  is  specified  in  part 
by  reference  2.  Further  detail  was  provided  through  review  of  local  policy 
and  interviews  with  test  officers  and  with  USAEPG's  Technical  Publications 
Division  personnel  responsible  for  preparing  and  publishing  test  plans. 

Subsequent  efforts  involved  acquisition  of  previously  drafted  boilerplate 
for  specific  subtests  and  review  of  Army,  AMC ,  TECOM,  and  USAEPG  publications 
to  refine  the  knowledge  of  test  plan  composition  and  of  the  overall  test  and 
evaluation  process  in  the  Army.  This  latter  knowledge,  in  addition  to  aiding 
m  understanding  the  test  planning  process,  is  essential  to  realize  the 
desired  training  benefits  from  use  of  the  intended  tool. 

2.2.  i. 3  Requirements .  The  general  requirements  taid  on  all  the  application 
efforts  selected  were  that  they  be  of  wide  utility  and  also  aid  in  training  of 
personnel.  Requirements  specific  to  TPD  were  that  it  reduce  the  manual  and 
telephonic  work  required  to  reach  the  strawman  stage  for  a  DTP,  that  it 
provide  information  on  test  plan  structure  and  component  descriptions ,  and 
that  it  assist  the  novice  test  officer  in  understanding  the  test  and 
evaluation  process.  Requirements  added  during  the  prototype  development  were 
that  it  assist  m  draft  DTP  preparation  and  ir.  the  mechanics  of  the  DTP  review 
process . 

A  requirement  of  the  TPD  infrastructure  was  that  it  be  maintainable  ir.  a 
form  accessible  to  a  broad  range  of  offices.  For  this  reason,  the  tod 
selected  to  create  and, maintain  these  components  should  be  one  that  is  wide. v 
available  and  understood  by  personnel  not  necessarily  involved  m  or  familiar 
with  expert  system  or  AI  development. 
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2.2. 1.4  Description.  The  current  TPD  prototype  performs  four  functions: 

a.  It  provides  a  simple  mechanism  for  entering  and  maintaining 
standard  information  pertinent  to  a  specific  test. 

b.  It  creates  a  partial  strawman  DTP  from  the  information  entered; 

c.  It  provides  background  information  on  the  test  and  evaluation 
process  in  general,  as  well  as  on  specific  components  of  a  test  plan. 

d.  It  provides  a  mechanism  for  preparing  a  draft  DTP  and  limited 
assistance  in  review  thereof. 

2.2. 1.5  Design/Development  Characteristics .  The  primary  development 
environment  selected  for  TPD  was  d3ASE  III.  This  environment  allows 
attainment  of  the  infrastructure  goals  without  having  to  retrofit  the 
infrastructure  at  a  later  time.  The  Al-related  tools  used  are  HYPE,  from 
Knowledge  Garden,  Inc.,  and  EXSYS  (Expert  System  Development  Package), 
obtained  from  California  Intelligence,  Inc.  HYPE  allows  use  of  the  hypertext 
paradigm  for  help  and  explanation  functions.  EXSYS  allows  development  of 
expert  system  components.  One  further  tool,  DOCUCOMP ,  from  Advanced  Software, 
Inc.,  provides  a  document  comparison  facility  for  identifying  changes  made  to 
a  standard  subtest  to  tailor  it  for  a  specific  system.  This  is  the  limited 
assistance  provided  in  draft  DTP  preparation  and  in  the  mechanics  of  the  DTP 
review  process  (section  2. 2. 1.3). 

The  initial  TPD  prototype  consists  entirely  of  the  dBASE  III  and  HYPE 
files  and  software  dealing  with  creation  and  maintenance  of  test  plan 
information,  and  associated  help  and  explanation  files.  The  dBASE  III 
application  software  drives  the  application,  invoking  HYPE  and  DOCUCOMP  where 
appropriate . 

Figure  2-1  shows  the  current  and  currently  planned  TPD  products  and  their 
status.  Figure  2-2  shows  the  overall  data  flow  for  the  application. 

2. 2. 1.6  Benef its/Ose .  TPD  is  designed  to  be  used  primarily  by  a  test  officer 
for  a  specific  system,  to  assist  in  preparing  a  strawman  DTP.  Another 
potential  user  is  the  manager  evaluating  a  proposed  test  outline  for  a 
potential  customer. 

The  current  TPD  prototype  is  installed  in  the  Command  and  Control 
Division  of  the  Command,  Control,  and  Communications  (Ca)  Test  Directorate. 

It  has  been  used  in  production  of  several  strawman  DTPs ,  and  the  current  users 
^ave  made  several  suggestions  for  improvement. 

The  most  obvious  benefit  to  be  gained  from  the  TPD  is  time.  Current 
users  and  others  to  whom  TPD  has  been  demonstrated  indicate  that  the  current 
manual  method  of  strawman  draft  plan  composition  can  take  from  two  days  to  two 
weeks.  The  TPD  output  is  available  within  15  to  30  minutes.  While  the 
resulting  product  is  not  complete,  it  accounts  for  perhaps  as  much  as  30  to  50 
percent  of  the  content  of  such  a  strawman.  Seme  increase  in  this  percentage 
will  accrue  from  growth  in  the  archive  of  standard  subtests,  while  some  must 
await  implementation  of  further  planned  functions. 
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Less  obvious  is  the  training  and  standardization  benefit.  The  hypertext 
provided  with  T?D  makes  available  to  the  novice  much  information  previously 
available  only  by  laborious  searching  through  assorted  regulations  and 
pamphlets.  It  also  indicates  sources  for  further  information.  Moreover, 
while  in  the  past  the  differing  backgrounds  and  levels  of  experience  among 
test  officers  have  sometimes  led  to  irregularities  in  DT?  and  subtest  format, 
more  widespread  use  of  a  single  tool  offers  the  promise  of  improved  adherence 
to  7ECCM  and  local  guidance  with  less  administrative  review  and  rewriting 
effort.  This  will  allow  test  officers,  test  engineers,  and  managers  to  devote 
more  of  their  time  and  effort  to  substantive  test  issues. 

2.2. 1.7  Development  Status.  T?D  prototype  functionality,  as  indicated  m 
figure  2-1,  is  roughly  70  percent  complete.  Addition  of  the  expert  system 
components  for  subtest  recommendations  and  coordination  requirements  will 
bring  the  system  to  a  levei  that  wiil  permit  initial  formal  evaluation  of  its 
utility.  This  will  require  making  the  tool  more  widely  available  to  test 
officers,  which  will  in  turn  require  additional  copies  of  supporting  software. 

2.2.  1.8  Future .  Given  that  the  TPD  proves  worthy  of  continued  development, 
three  major  lines  of  development  are  foreseen.  The  first  is  expansion  and 
refinement  of  the  knowledge-based  portion  of  the  system,  i.e.,  the  hypertext 
and  expert  system  components.  These  offer  considerable  potential  for 
expansion  into  expert  test  planning  areas,  such  as  costing  and  scheduling,  as 
well  as  tutorial  and  advisory  components  for  test  officer  training. 

The  second  area  involves  the  conventional,  infrastructure  component.  The 
knowledge  infrastructure  issue  is  documented  at  paragraph  2.1.4  above.  It  is 
important  here  to  note  only  that  this  effort  has  created  a  skeletal 
infrastructure  in  conventional  code  to  investigate  the  maintenance  and 
integration  problems  that  might  arise. 

The  third  line  of  development  involves  support  tool  integration.  The  use 
of  a  conventional  basis  for  this  tool,  and  development  of  a  standard  shell  for 
passing  the  information  contained  in  a  test-specific  data  base  to  an  expert 
system  component,  constitutes  an  example  of  one  integration  approach.  Further 
refinement  of  this  mechanism,  and  comparison  with  others,  is  essential  to 
allow  integration  of  the  support  tools  into  a  single  package  for  use  by  the 
vest  officer. 

2.2.2  Environmental  Impact  Assessment  Expert  System. 

2.2.2.  1  Fursose^Goals .  The  purpose  of  EVA  is  to  assist  the  test  officer  one 
environmental  personnel  in  collecting  accurate  environmental  information 
during  the  early  planning  phases  of  test  activities,  and  m  making  appropriate 
recommendations  based  on  characteristics  of  the  proposed  activities.  Specific 
goals  of  the  system  were  to: 

a.  Identify  tests  with  minimal  or  no  environmental  impacts,  and 
streamline  the  documentation  process. 


b.  Identify  possible  environmental  impacts  and  the  resources  that  couid 
be  affected  (e.g.,  water,  wildlife,  cultural,  historical). 

c.  Improve  the  quality,  detail,  and  timeliness  of  information  provided 
to  environmental  personnel  during  the  initial  stages  of  a  test  project. 

d.  Incorporate  environmental  information  into  the  initial 
a  >cision-makir.g  stages  of  a  project. 

e.  Guide  activity  proponents  through  the  environmental  assessment 
process,  ana  list  points  of  contact  for  action  items  and  regulatory 
requirements . 


; .  1 . 2  .  I  Do  mam/  Expertise, 
'equired  to 


The  domain  of  EVA  covers  that 


area  of  xr.c w. e cge 

identify  potential  environmental  impacts,  recognise  categorical 
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:e  rules  for  certain  damaging  activities,  ar.d  perform  a 
preliminary  screening  to  determine  the  probable  environmental  documentation 
requirements.  This  expertise  resides  with  the  USAEPG  environmental  quality 
coordinator  and  environmental  specialists  attached  to  the  post  garrison. 
These  experts  in  turn  consult  specialists  in  more  narrow  domains  when 
necessary . 


As  EVA  evolved  through  various  prototype  stages,  additional  information 
from  documented  sources  was  incorporated  into  the  design.  This  information 
consisted  more  of  quantitative  impact  factors,  rather  than  intuitive  knowledge 
about  the  domains.  The  inferences  about  this  data  were  supplied  by  the  human 
domain  experts. 


At  the  end  of  prototype  development  the  following  sources  had  been  used 
m  generating  the  data  bases  and  rules  of  EVA: 


a.  ’J.  S.  Army  Corps  of  Engineers 

1.  Construction  Engineering  Research  Laboratory  reports 

2.  Archaeologist 

b.  U.  3.  Department  of  Agriculture,  Soil  Conservation  Service 

c.  U.  3.  Army  Environmental  Hygiene  Agency 

d.  Fort  Huachuca 

1.  Forester 

2.  Wildlife  Biologist 

3.  Fish  Biologist 

4.  Environmental  Specialist 

Much  credit  is  due  the  po3t  environmental  specialist  for  identifying 
sources  of  information  and  eliciting  knowledge  from  subdomain  experts.  This 
effort  exceeded  the  scope  of  the  normal  participation  of  an  expert,  and  aided 
tremendously  in  knowledge  acquisition  activities. 

2. 2. 2. 3  Requirements .  USAEPG  is  required  to  conform  to  federal  and  state 
environmental  regulations  as  well  as  Army  and  DoD  policy  m  these  matters. 
Every  proponent  of  an  exercise  or  test  at  Fort  Huachuca  is  required  to  address 
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cers 


the  environmental  issues  associated  with  the  activity.  'JSAZPG  test  o  f :  i 
have  the  additional  responsibility  for  assessing  potential  environmental 
impacts  for  any  activity  resulting  from  a  test  directive,  regardless  of  the 
nature  of  the  testing. 

The  result  of  the  preliminary  imp  •ict.  <3ls s 6 s s n t<  is  &  record  oi 
environmental  consideration  ( REC )  .  The  ?.EC  documents  the  consideration  of 
environmental  impacts;  possible  outcomes  are  that  the  activity  is  adequately 
covered  by  existing  documentation,  qualifies  for  an  established  categorical 
exclusion  or  other  exemption,  or  requires  an  environmental  assessmer, t 
Environmental  assessments  subsequently  result  m  a  'finding  of  no  significant 
.mpact'  or  indicate  that  an  extensive  environmental  impact  statement  .3 
requi red . 


Most  of  USAEPG's  activities  are  conducted  at  locations  specifical 
designated  and  documented  for  that  type  of  activity,  or  are  conducted 
within  an  enclosed  facility,  such  as  computer  simulation  and  modeling, 
the  major  requirement  of  a  preliminary  environmental  screening  is  to 
discriminate  as  early  as  possible  between  typical  situations  requiring 
further  documentation,  and  those  requiring  a  significant  environmental 
study . 


entirely 


*  nus 


little 

impact 


2. 2.2. 4  Description .  The  EVA  elicits  information  about  a  proposed  activity 
from  the  test  officer,  and  reaches  a  preliminary  conclusion  on  the  actions 
required.  It  then  generates  a  report  containing  action  items,  and  summary  ana 
detail  characteristics  of  the  activity,  with  corresponding  environmental 
impacts.  Activities  which  have  already  been  documented  or  qualify  for  a 
categorical  exclusion  are  quickly  identified  (i.e.,  a  minimum  of  user  input  is 
required),  and  the  necessary  BEC  report  13  generated.  For  activities  where 
the  potential  environmental  impact  is  greater,  the  user  may  elect  to  examine 
the  environmental  resources  most  affected  and,  if  possible,  modify 
characteristics  of  the  proposed  activity  to  minimize  the  impact  and  associated 
documentation.  In  any  event,  information  from  the  report  is  used  by  the 
environmental  quality  coordinator  in  completing  the  environmental 
requirements . 

2. 2. 2. 5  Design/Development  Characteristics .  The  EVA  system  consists  of  an 
expert  system  which  provides  the  user  interface,  contains  the  rules  used  to 
make  decisions,  generates  reports,  and  interfaces  with  other  tools  for 
additional  capabilities.  These  other  tools  supply  such  functions  as  access  to 
data  bases  and  graphic  display  of  map  information.  Other  components  include 
supporting  information  such  as  help,  system  parameter,  map,  point-of -contact , 
and  report  specification  files.  The  expert  system  3hell,  EXSYS,  allows  a 
means  to  interface  with  the  other  tools  and  files  so  efficiently  that  the  user 
is  generally  unaware  of  the  individual  components.  To  further  isolate  the 
user  from  having  to  contend  with  directory  structures  and  operating  svstem 
commands,  a  set  of  command  files  was  created  to  simplify  the  installation  and 
operation  of  EVA.  The  user  merely  enters  one  command  to  run  the  system  arc 
display  and  print  the  results. 

The  main  expert  component  of  EVA  contains  about  120  IE-THEN  rules  in  the 
knowledge  base.  When  processed  by  the  EXSYS  inference  engine,  the  ru.es  serve 
to  collect  the  necessary  information  to  reach  the  final  conclusion  or.  the 
environmental  impacts  of  the  proposed  activity.  Forward  chaining,  a  technique 
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which  determines  how  the  ruie3  are  processed,  also  allows  some  control  over 
the  sequence  in  which  events  take  place.  Thus,  the  user  can  be  presented  with 
queries  in  the  same  relative  order,  even  though  the  knowledge  base  and 
supporting  data  bases  may  have  changed  from  previous  versions. 

Although  all  of  the  rules  nay  apply  to  a  given  scenario,  only  those  which 
rely  upon  unknown  information  will  request  the  user  to  enter  needed  data. 
Besides  background  information  such  as  project  number  and  description,  which 
are  always  requested,  firing  (processing)  of  the  rules  may  trigger  queries  on 
up  to  150  numerical  or  textual  variables,  and  up  to  35  multiple-choice 
questions.  For  example,  if  the  activity  will  include  aircraft,  then 
information  is  requested  on  the  number  of  aircraft,  number  of  flight  hours, 
and  time  of  day  and  altitude  of  the  flights.  Because  only  essential 
information  is  requrested,  ar.d  EVA  session  can  last-anywhere  from  5  to  45 

171 1  IK*  v  0  a  • 

Fart  of  the  development  philosophy  was  to  minimize  the  amount  of 
knowledge  to  be  included  in  the  rules  about  a  specific  installation. 

Information  on  the  location  of  sensitive  resources,  period  of  sensitivity  if 
not  constant  throughout  the  year,  and  qualitative  damage  factors  associated 
with  particular  activities,  were  placed  in  ten  data  bases.  These  data  bases 
were  designed  to  be  readily  understood  and  modified  by  the  domain  experts 
without  first  having  to  obtain  knowledge  engineering  skills.  Likewise,  user 
help  screens,  point-of -contact  information,  etc.,  which  contained 
installation-specific  material,  were  kept  in  separate  files.  This  approach 
may  provide  a  ready  means  of  porting  the  system  to  other  installations,  but 
was  chosen  primarily  to  reduce  development  and  maintenance  costs.  Information 
contained  in  the  various  data  bases  and  files  could  have  easily  been  encoded 
into  rules,  and  some  expert  development  packages  provide  the  capability  to  do 
just  that  when  fed  tabular  data.  The  problem  with  a  pure  expert  system 
solution,  with  all  of  the  knowledge  embedded  in  rules,  concerns  the  size  of 
the  resultant  rule  basec — It  was  estimated  that  to  incorporate  the  knowledge 
in  the  data  bases  alone  into  rules  would  add  another  three  to  four  hundred 
rules.  Further  development  and  maintenance  of  such  an  unwieldy  knowledge  base 
would  have  signi i icantly  impeded  progress,  with  no  known  advantages. 

2. 2. 2. 6  Be.nef  its/Use  ■  EVA  offers  benefits  to  the  test  officer,  environmental 
quality  coordinator,  and  program  managers.  Test  officers  are  given  the 
opportunity  to  compare  environmental  effects  of  different  activities  at 
various  locations  and  time3.  With  little  prior  knowledge  of  environmental 
concerns,  the  test  officer  using  EVA  can  quickly  gain  an  appreciation  of  the 
relative  impact  of  various  activities  through  the  questions  asked,  the 
associated  help  text,  and  the  outcome  of  the  proposed  scenarios.  Less 
experienced  test  officers  also  benefit  from  the  action  items  and  notes  related 
to  the  proposed  activity;  e.g.,  contacting  the  fire  marshal.  ar.d  filing  a  fire 
plan  if  incendiary  devices  are  used,  or  coordinating  tree  and  brush  removal, 
with  the^post  forester.  These  serve  as  reminders  even  for  seasoned  test 
officers,  and  both  inexperienced  and  experienced  users  of  the  system  benefit 
from  reduced  paperwork  and  coordination. 

EVA  does  not  make  complicated  environmental  decisions,  write 
environmental  assessments  or  environmental  impact  statements,  or  replace 
environmental  personnel.  In  fact,  environmental  quality  coordinators 
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themselves  can  use  EVA  to  refine  the  work  initiated  by  nest  officers,  or  a3  a 
method  of  automating  and  documenting  activities  in  a  standard  fashion.  Tests 
with  minimal  environmental  impact  are  identified  with  a  savings  of  paperwork 
and  time,  Even  for  large  activities  not  fully  handled  by  EVA,  the  quality, 
consistency,  and  detail  of  information  presented  to  environmental  personnel  is 
greatly  improved.  Without  EVA,  many  preliminary  meetings  are  required  between 
the  test  officer  and  environmental  quality  coordinator,  merely  to  establish 
what  information  is  needed,  and  then  the  data  is  rarely  available  in  an 
organised  format. 

Sponsors  of  testing  activities  may  gam  the  most  from  the  use  of  EVA, 
albeit  somewhat  indirectly.  Because  extensive  environmental  documentation 
requirements  can  cause  lengthy  and  expensive  delays,  it  is  important  to 
identify  potential  impacts  as  early  as  possible,  and  develop  alternative  test 
scenarios  which  are  more  environmentally  benign.  Advance  warning  of 
potentially  expensive  activities,  such  as  disposal  of  hazardous  materials 
(e.g.,  expended  batteries),  may,  if  given  m  time,  allow  implementation  of 
more  cost-effective  solutions. 

2. 2. 2.7  Development  Status.  EVA  is  currently  installed  on  several 
microcomputer  systems  at  Fort  Huachuca;  about  20  test  officers  have  been 
formaiiy  trained  in  its  use.  Presently  the  system  is  in  an  evaluation  phase, 
where  feedback  is  being  obtained  concerning  it3  use  in  test  operations. 

2. 2. 2. 8  Future .  A  number  of  ideas  for  further  development  of  EVA  have  been 
proposed.  During  its  construction,  the  development  team  identified  a  number 
of  desirable  features  which  could  not  be  implemented  because  of  time 
constraints.  Other  valuable  ideas  emerged  from  the  test  officer  training 
sessions.  However,  the  actual  usefulness  and  benefits  to  be  realized  must  be 
determined  from  the  results  of  the  ongoing  evaluation.  Some  of  the  more 
significant  limitations  and  improvements  to  be  considered  in  future  efforts 
are  the  following: 

a.  Some  of  the  knowledge  in  EVA  is  in  a  preliminary  state,  having  been 
added  to  determine  the  feasibility  and  desirability  of  certain  features  (e.g., 
a  component  to  address  hazardous  materials) .  Those  features  deemed  desirable 
should  be  expanded,  along  with  the  rest  of  the  system,  into  a  fully 
operational  form. 

b.  The  potential  for  porting  the  system  to  other  instai iations  should  be 
explored  further.  This  would  require  an  initial  analysis  of  the  requirements 
of  other  installations,  to  see  if  enough  commonality  exists  in  the  knowledge 
domains  to  make  this  approach  feasible.  Such  an  investigation  might  also  shed 
some  light  on  the  commonality  of  other  requirements,  such  as  test  resource 
management  and  safety. 

c.  The  prototype  system  has  the  limitation  that  only  one  map  area  can  be 
entered  as  the  location  of  activity.  Although  areas  may  be  arbitrarily 
defined  as  large  or  small  as  desired,  a  cumbersome  situation  occurs  with 
activities  consisting  of  100  or  more  sites  with  minimal  impact  at  each 
location.  Even  smaller  activities  may  be  handled  better  if  multiple 
locations,  or  if  unrestricted  boundaries  are  allowed. 


d.  A  feature  which  would  allow  saving  ail  of  the  input  information,  to 
be  used  later  to  examine  the  impact  of  different  test  scenarios,  is  desirable. 
Such  a  capability  was  partially  implemented,  but  had  to  be  disabled  because  of 
a  software  discrepancy  in  the  expert  system  tool.  Along  these  same  lines, 
many  users  expressed  the  desire  to  be  able  to  modify  an  entry  that  had  just 
been  made.  Both  seem  to  be  necessary  features  for  practical  use  in  an 
operational  environment. 

e.  Most  of  the  data  bases  of  EVA  are  indexed  by  location.  Geographic 
information  also  plays  an  important  part  in  many  other  functions  at  Fort 
Huachuca.  A  solution  to  many  of  these  needs  for  information  associated  with 
geographic  position  would  be  a  geographic  information  system.  This  is  also  a 
requirement  of  many  other  proposed  test  tools.  While  implementation  and 
maintenance  of  such  a  system  is  well -beyond -the  scope  of  this  investigation, 
the  potential  usefulness  is  great  enough  to  warrant  development  by  other 
means . 


f.  The  actual  users  of  EVA  range  from  inexperienced  test  officers  to 
qualified  environmental  personnel.  Because  of  the  disparity  in  experience,  a 
system  tailored  to  a  given  skill  level  will  be  somewhat  frustrating  for  users 
of  a  different  level.  Experienced  users  quickly  tire  of  a  system  oriented 
toward  the  novice,  while  inexperienced  users  may  find  a  system  written  for  the 
expert  to  be  much  too  difficult.  A  possible  solution  to  this  dilemma  was 
discovered  during  the  EVA  development,  but  too  late  to  fully  evaluate. 
Basically,  this  approach,  if  implemented,  would  call  for  multiple  levels  of 
rules,  help,  and  queries.  A  "don’t  understand"  option  is  provided  on  higher 
level  queries.  When  invoked  by  the  novice,  this  option  fires  lower  level 
rules  which  elicit  a  number  of  simpler  details  from  the  user.  These  details 
are  then  formulated  by  the  lower  level  rule3  into  facts  which  satisfy  the 
original,  "difficult"  query .  Such  an  approach  is  best  implemented  on  mature 
knowledge  bases  because  of  the  growth  in  size  and  commensurate  decline  in 
maintainability.  For  a  system  with  a  diverse  user  base,  further  examination 
of  this  technique  may  prove  useful. 

2.2.3  Meteorological  Expert  System. 

2.2.3. 1  Purpose/Goals .  The  Meteorological  expert  system  (MET)  began 
originally  as  a  manual  paper  checklist  for  test  officers  to  use  in  preparing 
for  upcoming  tests  at  Fort  Huachuca.  It  is  designed  to  emphasize  the  need  for 
meteorological  data  in  planning  and  reporting  tests  within  USAEPG.  MET  also 
indicates  that  various  meteorological  measurements  and  advisories  are 
available  from  the  Atmospheric  Sciences  Laboratory  (ASL)  weather  station  at 
Fort  Huachuca,  and  from  other  sites  located  on  the  Fort  Huachuca  ranges. 

2. 2. 3. 2  Domain/Expertise .  This  expert  system  deals  with  the  knowledge 
encompassing  meteorological  measurements  and/or  those  weather  events  which 
affect  test  operations  on  the  ground  or  in  the  atmosphere  where  testing  will 
take  place.  Generally  these  measurements  or  observations  are  provided  by  ASL. 

2. 2. 3. 3  Requirements .  From  the  standpoint  of  the  test  officer,  the  need  for 
an  expert  system  on  weather  is  that  it  can  educate  and  inform  the  test  officer 
about  meteorological  data  requirements  and  available  resources  for  a  test.  The 
need  for  such  data  comes  primarily  when  the  test  will  be  conducted  outdoors. 
The  expert  system  will  make  clear  that  the  officer  will  need  to  have  weather 
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predictions  before  the  test  in  order  to  plan  for  conditions  such  as  cold  or 
heat,  rain  or  snow,  and  wind  or  lightning.  Weather  advisories  and  weather 
alerts  from  ASL  can  warn  the  test  officer  in  the  field  of  impending  sudden 
weather  changes  that  could  endanger  personnel  and  equipment. 

2. 2. 3. 4  Description .  The  MET  system  educates  the  test  officer  as  to  possible 
weather-related  needs,  and  informs  the  officer  on  how  to  obtain  needed 
measurements  to  prepare  for  the  test,  how  to  run  the  test  more  effectively, 
and  how  to  obtain  weather  station  support  m  reporting  the  test  outcome. 


Measurements  and  predictions  of  temperature,  dew  point,  rain,  snow, 
thunderstorm  activity.,  and  winds  m  the  lower  atmosphere,  may  be  needed. 
Predictions  may  be  needed  as  to  meteorological  conditions  such  as  sunspot 
activity  and  atmospheric  index  of  refraction.  MET  informs  the  test  officer 
whether,  during  on-site  test  activities,  weather  advisories  and  reports  of 
selected  meteorological  values  are  available  and  nay  be  needed.  Also  the  ASL 
weather  station’s  ability  to  support  test  reporting  is  covered. 

The  result  of  using  the  MET  system  is  that  the  test  officer  can  produce 
better  test  data  by  being  prepared  with  needed  meteorological  data,  both  in 
measurements  that  directly  supply  parameters  needed  in  the  calibration  of 
equipment  such  as  radar,  and  in  supplying  measurements  for  the  test,  as  well 
as  weather  advisories  that  assist  in  day-to-day  running  of  test  operations. 


Without  MET,  the  test  officer  must  know  to  inform  ASL  of  test 
requirements  far  enough  in  advance  to  prepare  them  to  supply  information 
needed  for  the  test.  ASL  may  need  to  prepare  ahead  of  time  to  be  able  to  make 
measurements  during  the  test,  and  will  need  to  know  what  data  are  needed  for 
the  test  report.  ASL  can  supply  reports  of  the  meteorological  conditions  that 
existed  during  testing. 

2. 2. 3. 5  Des ign/Deve looment  Characteristics .  The  MET  system  is  composed  of  a 
senes  of  questions  which  are  presented  to  a  test  officer  from  within  the 
EXSYS  shell.  The  questions  asked  in  this  prototype  version  of  MET  determine, 
for  example,  whether  lasers  will  be  used  in  the  atmosphere,  whether  any  radar 
or  unmanned  aeronautical  vehicles  (UAV)  will  be  used,  whether  personnel  and/or 
equipment  will  be  in  the  field,  and  whether  heavy  rain  or  snow  will  be  a 
problem.  From  such  factors,  MET  can  then  advise  that  meteorological 
measurements  will  be  needed  to  support  these  activities.  For  example: 


a.  Aerosol  density  in  the  atmosphere  or  optical  scintillation 
measurements  may  be  needed  for  a  test  involving  lasers. 

b.  Meteorological  data  used  m  radar  calibration  may  be  needed  for  a 
test  using  or  testing  radar. 

c.  Measurements  of  upper  air  winds  and  turbulence  could  be  needed  for 
a  test  using  UAVs . 

d.  Weather  advisories  would  be  wise  to  have  during  test  activities. 


i  n 
hel 


MET 
which 
p  is 


automates  the  original  weather/meteorological  checklist  into  a  sy 
the  questions  are  presented  on  the  computer  monitor  for  decision, 
provided  by  way  of  a  computer-stored  text  file  for  each  question. 


stem 


and 
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the  answers  are  stored  in  computer  memory  until  the  sessions  end,  when  a 
report  including  all  input  answers  is  produced.  The  report  is  displayed  on  the 
computer  monitor  and  printed  on  the  line  printer,  under  operator  control. 


2. 2. 3. 6  3er.ei  i  ts/Use .  The  benefit  of  using  the  MET  system  is  that  the  test 
officer  becomes  better  informed  about  available  support  from  the  ASL  weather 
station,  and  learns  what  weather  conditions  require  special  preparation.  Tne 
test  officer  can  then  more  likely  plan  the  test  so  as  to  produce  a  more 
accurate  result,  and  will  be  able  to  write  a  more  correct  and  informative 
report.  This  all  adds  up  to  savings  in  time  and  money. 

2. 2.3.7  Development  Status.  MET  has  been  developed  only  to  the  initial 
evaluation  stage.  In  this  prototype  version,  MET  has  been  placed  c.n  10 
microcomputers  in  the  USAEPG  and  ASL- offices  at  Fort  Huachuca,  so  as  to  be 
available  for  use  by  all  test  officers.  Statistics  on  system  usage  and 
comments  on  deficiencies  or  possible  improvements  have  not  yet  been  collected. 

2. 2. 3. 8  Future .  After  evaluation,  the  MET  prototype  will  be  modified  to 
eliminate  any  discrepancies  found,  and  to  enhance  the  system's  capabilities  to 
better  serve  test  officer  needs.  Questions  will  be  improved  to  clarify  their 
meaning.  The  MET  help  file  will  be  changed,  as  needed,  to  make  explanations 
more  useful  to  the  user.  The  sequence  of  questions  presented  to  each  test 
officer  will  be  determined  by  previous  answers  to  prevent  redundancies. 

2.3  NEW  TECHNOLOGY  (NEUBAL  COMPUTING) 

Preliminary  efforts  on  this  investigation  focused  on  the  application  of 
AI  expert  system  technology  because  this  area  is  more  mature  than  other  AI 
techniques  and  technologies.  However,  rapid  progress  is  being  made  in  the 
other  areas.  The  next  major  technology  to  emerge  from  AI  laboratories  will 
probably  be  from  the  category  of  AI  known  as  neural  computing.  Since  the 
products  of  neural  computing,  neural  networks,  are  beginning  to  appear  in 
commercial  applications,  it  was  felt  that  this  aspect  of  AI  should  be 
investigated  further.  To  this  end,  an  initial  investigation  into  neural 
computing  was  performed  to  assess  the  feasibility  of  employing  neural  networks 
in  future  tools.  The  findings  of  this  effort  are  documented  in  volume  II  of 
this  report. 

2.4  TESTING  AI  ISSUES. 

In  dealing  with  issues  related  to  testing  of  Al-based  systems,  USAEPG  has 
addressed  them  at  two  levels.  On  the  one  hand,  USAEPG  has  been  faced  with  the 
problems  of  test  and  evaluation  of  its  own  expert  system  development  efforts. 
On  the  ether  hand,  it  has  attempted,  through  literature  reviews,  participation 
in  professional  exchanges  and  small  business  innovative  search  (SSI?:) 
efforts,  to  identify  or  encourage  research  ar.d  deveiepmeno  efforts  leading  to 
test  techniques  and  methodologies  for  such  systems.  In  both  cases,  the 
attention  has  been  directed  almost  entirely  towards  test  and  evaluation  of 
expert  systems,  as  they  are  the  most  widely  implemented  ar.c  most  highly 
evolved  of  Al-reiated  technologies. 


2-15 


2.4.1  In-House  Philosophy. 


As  indicated  in  reference  1,  well-founded  techniques  and  implementing 
technologies  for  test  and  evaluation  of  expert  systems  are  virtually 
nonexistent.  While  the  past  year  has  witnessed  the  appearance  of  a  number  of 
prototype  techniques,  and  an  even  wider  range  of  proposals,  none  of  these  is 
at  the  stage  of  development  which  would  allow  acquisition  of  implementing 
software  tools  for  generic  systems.  For  this  reason,  USAEPG’ s  test  and 
evaluation  of  its  own  development  efforts  have  been  limited  to  expert  and 
developer  'walk-throughs'  of  the  respective  knowledge  bases,  and  direct 
comparison  of  human  expert  and  expert  system  performance  on  test  cases. 

For  mc.t  of  the  systems  developed  by  USAEPG,  limitation  to  these  test  and 
evaluation  techniques  has  been  less  problematic,  m  that  most  of  the  systems 
involved  have  been  developed  in  large  part  from  domain  knowledge  in  published 
form.  Thus  much  of  the  organization  into  a  readily  assimilable  form  has 
already  been  accomplished,  including  examples  or  exercises  which  may  serve  as 
initial  test  cases,  making  both  development  and  review  much  simpler. 

2.4.2  Test  Items  with  AI ■ 

USAEPG  has  a  long-term  goal  of  identifying  and  investigating  techniques 
applicable  to  test  items  containing  Al-based  components.  Current  efforts  have 
been  directed  at  techniques  applicable  to  expert  systems  which  may  appear  as 
embedded  software  in  battlefield  automated  systems  (BAS). 

Four  techniques  are  in  use  at  present  and  have  3ome  potential  for 
application  to  BAS.  These  include: 

a.  Developer/expert  review  (walk-through)  of  the  knowledge  base. 

b.  Comparison  of  system  performance  against  human  expert  or  expert 
panel  performance  on  a  set  of  test  cases. 

c.  Comparison  of  system  performance  against  that  of  an  existing 
conventional  system  or  technique. 

d.  Prediction  of  desirable  or  undesirable  system  characteristics 
from  analysis  of  the  knowledge  base  and  inference  method,  and  testing  for  the 
predicted  characteristics . 

As  indicated  above,  the  first  and  second  techniques  have  been  used  to 
assess  the  performance  of  systems  developed  by  USAEPG.  The  second  and  fourth 
techniques  were  use'1  to  a  limited  extent,  in  an  observation  of  a  prototype 
expert  system  developed  by  the  Army  under  the  Very  Intelligent  Surveillance 
and  Target  Acquisition  program.  All  of  the  techniques  have  been  applied  by 
one  or  more  organizations  and  found  useful,  although  they  are  all  manpower 
intensive  and  involve  dedication  of  highly  skilled  personnel  for  their 
application . 

A  number  of  research  efforts  are  under  way  to  create  automated  aids  for 
checking  various  characteristics  of  a  knowledge  base  at  both  ioca..  and  global 
levels.  While  most  of  these  are  directed  at  creation  of  sophisticated 
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development  aids,  their  utility  in  a  testing  environment  would  be  immense. 

The  greatest  impediment  to  applying  these  techniques  to  expert  systems 
embedded  in  BAS  is  the  lack  of  standards  of  knowledge  representation, 
inference  mechanism  definition,  and  actual  implementation  representation .  The 
tools  being  developed  are  targeted  for  a  narrow  range  of  development 
environments ,  and  the  portability  of  the  techniques  is  an  open  issue.  To 
address  this,  USAEPG  has  submitted  SBI3  proposals  to  investigate  creation  of  a 
taxonomy,  to  include  standard  definitions  of  expert  system  technologies ,  as 
well  as  to  investigate  a  standard  for  expert  system  knowledge  base 
representation.  In  addition,  an  SBIE  phase  II  effort  is  now  under  way, 
examining  existing  and  proposed  expert  system  test  techniques  in  some  detail. 

In  order  to  foster  greater  cooperation  among  researchers  in  this  area, 
and  to  bring  together  some  af  the  existing  efforts,  a  USAEPG  representative 
acted  as  cochairperson  of  a  workshop  on  test  and  validation  of  expert  systems, 
which  was  held  at  the  annual  AAAI  conference  in  August  1988  in  St.  Paul, 
Minnesota.  Over  forty  researchers  attended,  and  some  twenty  presented  papers. 
The  full  formal  proceedings  of  the  workshop  are  in  preparation.  In  addition 
to  allowing  new  points  of  contact,  the  workshop  was  responsible  for  several 
new  coordinated  efforts,  bringing  together  researchers  not  previously  familiar 
with  one  another’s  work. 

As  a  result  of  the  workshop,  USAEPG  has  established  direct  contact  with  a 
number  of  the  principle  researchers  investigating  test  and  validation  of 
expert  systems,  as  well  with  the  other  organizations  funding  such  research. 

The  proceedings  will  provide  a  snapshot  of  the  state  of  the  art  at  this  point 
in  time,  and  an  independent  check  to  apply  to  the  results  of  the  SBIR  efforts 
when  those  are  received. 


This  page  intentionally  blank 


2-18 


SECTION  3 .  APPENDIXES 


3-1 


This  page  intentionally  blank 


3-2 


APPENDIX  A,  METHODOLOGY  INVESTIGATION  PROPOSAL 


28  August  1987 

METHODOLOGY  INVESTIGATION  PROPOSAL 


1.  TITLE .  AI  Test  Officer  Support  Tool 

2.  INSTALLATION  OR  FIELD  OPERATING  ACTIVITY.  US  Army  Electronic  Proving 
Ground,  Fort  Kuachuca,  Arizona  85613-7110. 

3.  PRINCIPAL  INVESTIGATOR.  Mr.  Robert  Harder,  Software  and  Automation 
3ranch  ,  STEEP-MT-DA,  AUTOVON  879- 1879/ IS70 . 

A,  BACKGROUND .  Test  design  and  planning  for  modern  C’l  systems  require 
familiarity  with  a  number  of  test  operating  procedures  (TOPsl  as  well  as 
detailed  knowledge  of  specific  test  tool  capabilities.  A  wide  variety  of 
tests  must  be  designed,  planned,  and  scheduled  m  order  to  efficiently  conduct 
testing.  Interrelationships  among  test  groups  and  tools,  common  data 
requirements  and  data  reduction  and  analysis  requirements,  lead  time  to 
prepare  instrumentation,  and  required  availability  of  the  test  item  must  be 
well  understood  in  order  to  efficiently  conduct  tests  within  allocated  time 
constraints . 

USAEPG  has  explored  the  feasibility  of  an  automated  system  to  support  the 
test  officer.  Using  Independent  Laboratory  In-house  Research  (ILIR)  funds,  a 
prototype  system  was  developed  using  AI  technology.  The  prototype  addressed 
tests  performed  by  the  Simulation  and  Interference  Branch  primarily,  but  could 
be  expanded  for  other  test  areas. 

5.  PROBLEM.  Testing  C3I  systems  involves  designing  and  planning  tasks  which 
are  becoming  increasingly  complex.  Advances  in  technology  such  as 
microprocessor  design,  distributed  reai-tirae  architectures,  artificial 
intelligence,  and  electro-optics  are  appearing  in  new  C’l  developments.  While 
this  sophisticated  technology  offers  benefits  to  the  developer,  it  is  becoming 
a  considerable  burden  to  the  tester.  Test  officers  are  required  to  identify 
appropriate  test  methods  and  associated  instrumentation  and  data  acquisition 
requirements  for  each  emerging  technological  area.  This  requires  a  level  of 
expertise  which  is  rarely  found  in  any  one  individual.  Besides  being 
distributed  among  individuals,  and  therefore  less  available,  this  hard-earned 
expertise  is  frequently  lost  to  the  organization  because  of  personnel 
reassignment  or  attrition. 

6.  OBJECTIVE .  To  improve  test  methodology  by  providing  the  test  officer  with 
an  automated  support  tool. 

7-  MISSION  AREA(S)  SUPPORTED.  All  DA  mission  areas  for  systems  containing 
embedded  computer  resources  (ECR)  are  supported.  The  'Big  5‘  program 
categories  (C3,  RSTA,  etc.)  are  accommodated  by  the  nor.system-speci  f  ic  nature 
of  the  methodology. 
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AI  Test,  Officer  Support  Tool  Cent) 
8.  PROCEDURES. 


a.  Summary.  The  investigation  will  draw  upon  previous  ILIR  efforts  by 
expanding  the  level  of  detail  and  the  scope.  The  result  will  be  an  enhanced 
tool  supporting  the  test  officer  in  specific  domains  such  as  electromagnetic 
compatibility,  software  testing,  and  general  test  mechanisms.  Other  domain 
categories  will  be  explored  as  time  permits. 

b.  Detailed  Approach.  The  USAEPG  will: 

( 1 )  Extract  and  codify  knowledge  from  cognisant  individuals  m  fields 
including  electromagnetic  and  software  testing. 

l 2 )  Examine  other  test  areas  to  identify  tests  performed,  responsible 
branches,  test  instrumentation  capabilities,  and  characteristic  test 
requirements.  Commonality  among  these  various  factors  will  be  identified  to 
form  a  framework  which  will  accommodate  all  test  functions,  instrumentation, 
and  resources.  Following  implementation  of  the  generalized  framework, 
specific  test  areas  (knowledge  domains)  will  be  analyzed  in  depth  and 
incorporated  into  the  tool. 

c.  Final  Product ( s) . 


(1)  An  AI  test  officer  support  tool  with  enhanced  capabi 1 l ty- -more 
'smarts'  m  the  existing  area  of  coverage,  and  additional  test  areas  covered. 

(2)  Requirements  and  recommendations  for  automation  of  test  design 
and  planning  functions. 

d.  Coordination.  Extensive  coordination  with  the  various  test  groups  of 
the  USAEPG  is  an  inherent  characteristic  of  the  investigation.  To  the  extent 
that  test  areas  covered  overlap  the  areas  of  interest  of  other  I/FOAs, 
coordination  will  be  accomplished  through  existing  mechanisms  such  as  the 
TECOM  Software  Technical  Committee  (TSOTEC) . 

e.  Environmental  Impact  Statement.  Execution  of  this  task  will  not  have 
an  adverse  impact  on  the  quality  of  the  environment. 

f.  Health  Hazard  Statement.  Execution  of  this  task  will  not  involve 
health  hazards  to  personnel. 

9.  JUSTIFICATION/IMPACT 

a .  Association  with  Mission. 

USAEPG’s  mission  relative  to  test 
with  automated  support  tools  will 
testing . 

b. .  Association  with  Methodoiogy/Instrumentation  Program.  This  project 
supports  thrusts  of  the  TECOM  Methodology  Program  to  improve  the  quality  of 
testing  as  well  as  test  process. 

c .  Present  Capability,  Limitations,  I 
net  Approved. 


This  investigation  directly  supports 
and  evaluation.  Providing  test  officers 
improve  the  efficiency  and  ef f ectiveness  c 


mprovement,  and  Impact  on  Testing  :f 


til  ‘O 


Tool  (Con*. 


AI  Te3*v  Officer  Support 


il)  Presen*  Capability.  TOPs  and  guide 
Officers  Handbook,  provide  static  information  on 
for  te3t  plann.ng  purposes. 


ir.es,  such  as  the  USAEP'3  Test 
test  methods  and  checklists 


i2)  Limi tations .  Current  guidelines  often  do  not  provide  the  level 
of  detail  required  for  optimized  application  of  scarce  test  resources.  Also, 
the  information  is  static;  status  of  test  instrumentation,  competition  for 
resources  among  different  test  items,  and  the  impact  of  not  performing  some 
test  (or  lack  of  test  material  such  as  certain  documentation)  is  poorly 
handled  unless  the  test  officer’s  experience  has  included  similar  situations. 

(3)  Improvement .  Using  AI  techniques  to  develop  a  support  tool  car. 
rovide  the  test  officer  sufficiently  detailed  and  flexible  guidelines, 
eside  being  adaptable  to  the  needs  of  a  specific  test  item  and  current  with 

respect  to  test  instrumentation  availability,  the  proposed  approach  would  be 
sensitive  to  data  requirements  and  be  able  to  anticipate  the  impact  if  tests 
are  not  performed.  Supported  over  time,  such  a  tool  could  accumulate 
expertise  which  is  presently  distributed  and  too  frequently  lost. 

(4)  Impact  on  Testing  if  not  Approved.  The  expertise  required  of 
test  officers  is  rapidly  expanding  in  scope  as  innovative  technologies  are 
increasingly  employed  by  developers.  The  corresponding  increase  in  complexity 
of  test  methods  and  instrumentation  demands  a  commensurate  improvement  in 
support  tools  if  test  resources  are  to  be  effectively  and  efficiently  used. 
Also,  without  permanent  storage  and  readily  available  access  to  "lessons 
learned' ,  the  corporate  memory  of  an  activity  suffers  each  time  an  experienced 
individual  leaves  the  organization. 

10.  D0LLA3  SAVINGS ■  No  directly  supportable  dollar  savings  can  be  projected 
at  this  time.  Indirect  benefits  include  improving  the  quality  of  testing  and 
evaluation  leading  to  improved  quality  of  fielded  systems.  Equally  difficult 
to  quantify  is  the  retention,  concentration,  and  increased  availability  of 
expertise,  which  is  potentially  a  significant  amount. 
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1 1 .  RESOURCES. 

a .  Financial . 


Dollars  (Thousands) 
FY68 

Dollars  (Thousands) 
FY8S 

In-House  Out-of-House 

In-House  Out-cf -House 

Personnel  Compensation 

10.0 

12.0 

Travel 

3 . 5 

4  . : 

Contractual 

Support 

84.5 

42  .  5 

Materials  Sc 

Suppl ies 

2 . 0 

1  .  5 

^ubtc  j a i s 

15.5 

84.5 

17.2  42.2 

FY  Totals 

100.0 

60.0 

b •  Explanation  of  Cost  Categories. 

(1)  Personnel  Compensation.  This  cost  represents  compensation 
chargeable  to  the  investigation  for  using  technical  or  other  civilian 
personnel  assigned  to  the  investigation. 

(2)  Travel .  This  represents  costs  incurred  while  visiting 
government  and  industry  facilities. 

(3)  Contractual  Support.  Performance  of  the  investigation  will  be 
accomplished  with  resources  provided  under  an  existing  support  contract. 

c .  Obligation  Plan  (FY89) . 

FQ  i  2  3  4  Total 

Obligation  Rate  45.0  5.0  5.0  5.0  50.0 

(Thousands ) 

d .  Man-Hours  Required. 


In-House : 
Contract : 
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1 2  .  ASSOCIATION  WITH  TOP  PROGRAM.  This  investigation  will  net  directly 
produce  a  TOP.  However,  various  TOPs  may  require  review  and  revision  bas 
the  findings. 

FOR  THE  COMMANDER: 


(signed) 

ROBERT  E.  REINER 
Chief,  Modern! cation  ar.d 
Advanced  Concepts  Division 


ed  on 
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APPENDIX  B. 


P.EFEHENCE5 

1.  Methodology  Investigat ion  for  Software  Technology  for  Adaptable,  Reliable 
Systems  (STARS),  Final  Report:  Test  Methodology  with  Artificial  Intelligence, 
Harder,  et  al . ,  U.S.  Army  Electronic  Proving  Ground,  Fort  Huachuca.  Arizona, 
March  1967. 

2.  TECOM  Regulation  70-24,  Documenting  TECOM  Testing,  22  June  1981. 


Note:  Domain-specific  references  are  available  upon  request. 
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NDIX  C.- 


ACRONYMS  AND  ASREZVI AT IONS 


AAAI .  American  Association  for  Artificial  Intelligence 

AI .  Artificial  Intelligence 

AMC .  United  States  Army  Materiel  Command 

ASL .  Atmospheric  Sciences  Laboratory 

BAS .  Battlefield  Automated  System 

C3 .  Command,  Control,  and  Communications 

C’l .  Command,  Control,  Communications  and  Intelligence 

DoD .  Department  of  Defense 

DT? .  Detailed  Test  Plan 

ES  = .  Expert  System  Selector 

EVA .  Environmental  Impact  Assessment  Expert  System 

EXSYS .  Expert  System  Development  Package 

I/FOA .  Installation  Field  Operating  Activity 

MET .  Meteorological  Expert  System 

HEC .  Record  of  Environmental  Consideration 

SBIR .  Small  Business  Innovative  Research 

STARS .  Software  Technology  for  Adaptable,  Reliable  Systems 

TECOM .  United  States  Army  Test  and  Evaluation  Command 

TPD .  Test  Plan  Drafter 

TSOTEC .  TECOM  Software  Technical  Committee 

UAV .  Unmanned  Aeronautical  Vehicle 

USAEPG .  United  States  Army  Electronic  Proving  Ground 
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APPENDIX  D.  DISTRIBUTION 
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Director 

US  Army  Materiel  Systems  Analysis  Activity 
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Commander 

US  Army  Test  and  Evaluation  Command 
ATTN:  AMSTE-TA-W 
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AMSTE-TA 
AMSTE-SI 

Aberdeen  Proving  Ground,  MD  21005-5055 
Administer 

Defense  Technical  Information  Center 
ATTN:  DDA 
Cameron  Station 
Alexandria,  VA  22304-6145 

Commander 

US  Army  Cold  Regions  Test  Center 

ATTN:  STECR-TM 

APO  Seattle,  WA  98733-5000 

Commander 

US  Army  Combat  Systems  Test  Activity 
ATTN:  STECS-DA-M 

Aberdeen  Proving  Ground,  MD  21005-5000 
Commander 

US  Army  Dugway  Proving  Ground 
ATTN:  STEDP-PO-P 
Dugway,  UT  84022-5000 

Commander 

US  Army  Electronic  Proving  Ground 
ATTN:  STEEP-TD 
STEEP-ET 
STEEP-CT 
STEEP-MO 

Fort  Huachuca,  AZ  85613-7110 
Commander 

US  Army  Jefferson  Proving  Ground 
ATTN:  STEJP-TD-E 
Madison,  IN  47250-5000 


Addressee 


Number 
of  Copies 


Commander 

US  Army  Tropic  Test  Center 

ATTN:  STETC-TD-AB  1 

APO  Miami,  FL  34004-5000 

Commander 

US  Army  White  Sands  Missile  Range 

ATTN:  STEWS-TE-A  1 

STEWS -TE-M  1 

STEWS-TE-0  1 

STEWS -TE-PY  4 

White  Sands  Missile  Range,  NM  88002-5000 

Commander 

US  Army  Yuma  Proving  Ground 

ATTN:  STEYP-MSA  2 

Yuma,  AZ  85634-5000 
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